Parallel cucumbers #7309

MarcelloPerathoner · 2025-12-22T08:45:02Z

Issue

Fixes #7307 - CI runs up from 0:45h to 4:30h

Improvements

Parallel execution of cucumber tests tremendously improves running times.

Note: when using --parallel N make sure there are N contiguous free ports at the
configured port number (eg. at ports 5000--5000+N).

A lot more can be configured with profiles (in cucumber.mjs). Usage examples:

$ npx cucumber-js -p home -p ch -p mmap --format html:/tmp/testlog.html --parallel 8
$ npx cucumber-js -p github -p mld -p directly --parallel 8
$ npm test -- --parallel $JOBS --fail-fast

Github step summaries display the test results:

Screenshot 2026-01-08 at 21-33-21 WIP Parallel cucumbers · Project-OSRM_osrm-backend@6b55220

On github CI full logs, also containing the outputs from all binaries, are published to
https://reports.cucumber.io/.

Code refactored and simplified. ~150 LoC less.

Other Fixes

During development other problems were found and fixed:

The datastore load method now works as expected: one and the same osrm-routed will
be running in the background for the whole duration of the test. Previous behaviour: for
each scenario an osrm-routed would be spawned, loaded with datastore, and then killed
again.
The data watchdog in osrm-routed now will emit the message: 'updated facade' right
after the update (previous behaviour: right before the update), so that it can be used
for process synchronization.

Breaking changes

If you run the tests using the npx cucumber-js ... calling method you must add either
-p home or -p github as the first profile, eg.:

$ npx cucumber-js -p home -p mld -p mmap --parallel 8

instead of:

$ OSRM_LOAD_METHOD=mmap npx cucumber-js -p mld

Old cached files are not cleaned automatically anymore because that interfered with
concurrency. There is no hook in cucumber that runs before the workers start.

We don't support changing the load method in mid-run anymore. (eg. "Given data is loaded
with datastore" no longer works). Reason: With the datastore load method an
osrm-routed will be constantly running in the background. No second osrm-routed can
open the same port again.

Retired Environment Variables

OSRM_CONNECTION_RETRIES
OSRM_CONNECTION_EXP_BACKOFF_COEF

Other changes

Removed one cache level. CH and MLD are processed and cached separately.

The outputs of osrm binaries are not written into lots of small files anymore. Instead
use the --format html:test/logs/testlog.html commandline argument to get one big HTML
file with all outputs embedded.

New configuration variables

New environment variables and/or cucumber worldParameters entries were introduced. The
whole configuration is done in cucumber.mjs.

Environment Variable	worldParameters	Defaults to
`CUCUMBER_TIMEOUT`	`timeout`	5000	Scenario timeout in ms.
`CUCUMBER_HTTP_TIMEOUT`	`httpTimeout`	2000	HTTP timeout in ms.
`CUCUMBER_TEST_PATH`	`testPath`	`test`	The test directory
`CUCUMBER_PROFILES_PATH`	`profilesPath`	`profiles`	The profiles directory
`CUCUMBER_LOGS_PATH`	`logsPath`	`test/logs`	The logs directory
`CUCUMBER_CACHE_PATH`	`cachePath`	`test/cache`	The cache directory
`OSRM_BUILD_DIR`	`buildPath`	`build`	Path to the binaries
`OSRM_LOAD_METHOD`	`loadMethod`	datastore	Data load method
`OSRM_ALGORITHM`	`algorithm`	ch	Routing algorithm
`OSRM_IP`	`ip`	127.0.0.1	IP Address
`OSRM_PORT`	`port`	5000	IP Port

New tags

Tag	A scenario thus tagged ...
`@isolated`	will not run while any other scenario is running in parallel
`@with_(datastore\|directly\|mmap)`	will be executed iff the load method matches
`@no_(datastore\|directly\|mmap)`	will be executed unless the load method matches
`@with_(ci\|mld)`	will be executed iff the algorithm matches
`@no_(ci\|mld)`	will be executed unless the algorithm matches

Tasklist

CHANGELOG.md entry (How to write a changelog entry)
TODO: clean up and remove some unrelated stuff
update relevant Wiki pages
review
adjust for comments

Requirements / Relations

Link any requirements here. Other pull requests this PR is based on?

afarber · 2025-12-27T09:03:42Z

Thanks for working on speeding up the CI! The parallel cucumber approach looks promising.

One concern: could we keep the CI matrix changes out of this PR? Switching from macos-13 to macos-15-intel drops test coverage for users on Ventura/Sonoma, and adding clang-20 (which isn't released yet) seems unrelated to the parallelization work. These changes could be discussed separately if there's a reason to drop older platform support.

Also spotted a few bugs in osrm_loader.js - there are references to an undefined err variable in the error handlers (lines 45, 118, 132) that would crash if osrm-routed fails.

MarcelloPerathoner · 2025-12-27T11:14:52Z

The PR is still WIP. I will cleanup before releasing it.

BTW clang-21 is current. The reason to drop macos-13 is that github does not support it anymore.

TheMarex · 2026-01-07T10:05:27Z

features/lib/osrm_loader.js

-        );
-        callback();
+    });
+    // process.on('SIGIO', this.resolve);


TheMarex · 2026-01-08T13:02:11Z

Do you think the non-cucumber changes (datastore, routed, etc) can be broken out independently without messing with the tests? It is always quite the challenge to review large changes like this so I would appreciate the extra effort if possible. 🙏

MarcelloPerathoner · 2026-01-08T13:27:10Z

The shared memory changes could easily go elsewhere, as they are not the primary reason for sync loss between the test suite and routed. But the change in routed is critical (and trivial too).

The changes to the github workflow should go elsewhere. I'm just playing around to get windows to run the test suite too. See if any surprises come up.

MarcelloPerathoner · 2026-01-08T22:37:40Z

Ready for review. I have removed everything that is not strictly necessary. The shared memory fixes for macOS and Windows will go into another PR. Until that macOS tests with datastore will be flaky.

features/support/env.js

docs/cucumber.md

features/support/cache.js

features/lib/osrm_loader.js

afarber · 2026-01-14T16:12:20Z

Looks like your PR #7321 resolved the root cause - nice work.

I notice you've added back setDefaultTimeout() with CUCUMBER_TIMEOUT support, but since all the CUCUMBER_TIMEOUT: 60000 entries were removed from the workflow, the env var is never set. It will always default to 5000ms, which is also Cucumber's built-in default.

You could simplify by removing setDefaultTimeout() again.

MarcelloPerathoner · 2026-01-14T16:26:46Z

Thanks. I put the timeout back in case somebody might want it badly, though I strongly advise against using it. Currently everything works fine with the default timeout.

TheMarex

Looks great. 🙇 Tried it locally and while I'll miss the never ending row of green dots broken by the inevitable red F this is so nice. Massive amount of work, appreciate it. 🙌 Only minor questions / comments.

TheMarex · 2026-01-15T16:48:57Z

features/options/datastore/datastore.feature

@@ -1,33 +1,63 @@
-@datastore @options @help
+@datastore @options @help @isolated @no_datastore


Hm I'm not so I follow: What does @datastore in conjunction with @no_datastore do?

@datastore is a legacy tag, means: run all test concerning osrm-datastore. Those tests do ugly things like: osrm-datastore --spring-clean. So they cannot run while other tests are actually using the datastore. @no_datastore is a new tag, meaning: don't run if the selected load method is datastore. A bit confusing ...

TheMarex · 2026-01-15T16:51:15Z

features/step_definitions/data.js

 });

 Given(/^the node map$/, function (docstring, callback) {
-  const q = d3.queue();


Hm interesting - any idea why this used to be async?

Because it was easier to call it this way instead of thru a chain of callbacks?

TheMarex · 2026-01-15T17:00:47Z

features/support/route.js

+    let query = '';
+    if (params.coordinates !== undefined) {
+      // FIXME this disables passing the output if its a default
+      // Remove after #2173 is fixed.


So actually that referenced issue is fixed. Though honestly it may be fine to leave it implicitly as is.

TheMarex · 2026-01-15T17:01:58Z

features/testbot/alternative_loop.feature

@@ -1,4 +1,4 @@
-@routing @testbot @alternative
+@routing @testbot @alternative @isolated @todo


Why the todo here?

That test must be rewritten. The result is unstable.

TheMarex · 2026-01-15T17:05:09Z

features/testbot/load.feature

+@routing @load @testbot @todo @isolated @no_datastore
 Feature: Ways of loading data
+
+# TEST BROKEN: we don't support changing the load method in mid-run.


If we don't start a new osrm-routed does this work though? The core here is I think the spring clean.

The logic for that test has been removed. While it may be possible to switch between mmap and directly I don't see the point. Most of those tests are duplicates.

TheMarex · 2026-01-15T17:05:46Z

scripts/cucumber_test_matrix.sh


 set -o errexit
 set -o pipefail
-set -o nounset


Left-over from debugging?

$GITHUB_STEP_SUMMARY may be undefined.

MarcelloPerathoner changed the title ~~WIP: Speed up cucumber tests~~ WIP: Parallel cucumbers Dec 24, 2025

TheMarex reviewed Jan 7, 2026

View reviewed changes

features/lib/osrm_loader.js Outdated

);

callback();

});

// process.on('SIGIO', this.resolve);

Copy link

Member

TheMarex Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left-over?

TheMarex mentioned this pull request Jan 8, 2026

Disable debug builds for now #7318

Merged

MarcelloPerathoner force-pushed the Fix-cucumber-routed-startup-timeout branch 2 times, most recently from e09c32e to 6b55220 Compare January 8, 2026 20:23

MarcelloPerathoner changed the title ~~WIP: Parallel cucumbers~~ Parallel cucumbers Jan 8, 2026

This was referenced Jan 9, 2026

osrm-datastore on macOS and win do not wait for clients to switch #7320

Closed

Fix macOS and Windows shared mem wait for detach #7321

Merged

Parallelize cucumber tests using GitHub Actions matrix #7319

Draft

MarcelloPerathoner force-pushed the Fix-cucumber-routed-startup-timeout branch from db91c60 to 0f33530 Compare January 10, 2026 21:51

afarber suggested changes Jan 12, 2026

View reviewed changes

features/support/env.js Show resolved Hide resolved

docs/cucumber.md Outdated Show resolved Hide resolved

features/support/cache.js Outdated Show resolved Hide resolved

features/lib/osrm_loader.js Show resolved Hide resolved

MarcelloPerathoner closed this Jan 14, 2026

MarcelloPerathoner force-pushed the Fix-cucumber-routed-startup-timeout branch from 0f33530 to 1e1d5fa Compare January 14, 2026 07:33

MarcelloPerathoner reopened this Jan 14, 2026

TheMarex approved these changes Jan 15, 2026

View reviewed changes

MarcelloPerathoner added 8 commits January 16, 2026 10:31

Rebase on origin/master

e9c08f8

WIP

67547ce

WIP

c8848dc

WIP

9c39fef

WIP

1bbd184

Include suggestions by afarber

59779e6

WIP

2118139

WIP

860d40d

WIP

c765fee

MarcelloPerathoner force-pushed the Fix-cucumber-routed-startup-timeout branch from eb6f405 to c765fee Compare January 16, 2026 09:34

		@@ -1,33 +1,63 @@
		@datastore @options @help
		@datastore @options @help @isolated @no_datastore

		@@ -1,4 +1,4 @@
		@routing @testbot @alternative
		@routing @testbot @alternative @isolated @todo

Parallel cucumbers #7309

Are you sure you want to change the base?

Parallel cucumbers #7309

Uh oh!

Conversation

MarcelloPerathoner commented Dec 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue

Improvements

Other Fixes

Breaking changes

Retired Environment Variables

Other changes

New configuration variables

New tags

Tasklist

Requirements / Relations

Uh oh!

afarber commented Dec 27, 2025

Uh oh!

MarcelloPerathoner commented Dec 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TheMarex commented Jan 8, 2026

Uh oh!

MarcelloPerathoner commented Jan 8, 2026

Uh oh!

MarcelloPerathoner commented Jan 8, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

afarber commented Jan 14, 2026

Uh oh!

MarcelloPerathoner commented Jan 14, 2026

Uh oh!

TheMarex left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

MarcelloPerathoner commented Dec 22, 2025 •

edited

Loading